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Abstract — Learning to operate a vehicle is generally ac- 
complished by forming a new cognitive map between the 
body motions and extrapersonal space. Here, we consider the 
challenge of remapping movement-to-space representations 
in survivors of spinal cord injury, for the control of powered 
wheelchairs. Our goal is to facilitate this remapping by 
developing interfaces between residual body motions and 
navigational commands that exploit the degrees of freedom 
that disabled individuals are most capable to coordinate. We 
present a new framework for allowing spinal cord injured 
persons to control powered wheelchairs through signals 
derived from their residual mobility. The main novelty of 
this approach lies in substituting the more common joystick 
controllers of powered wheelchairs with a sensor shirt. This 
allows the whole upper body of the user to operate as an 
adaptive joystick. Considerations about learning and risks 
have lead us to develop a safe testing environment in 3D 
Virtual Reality. A Personal Augmented Reality Immersive 
System (PARIS) allows us to analyse learning skills and pro- 
vide users with an adequate training to control a simulated 
wheelchair through the signals generated by body motions 
in a safe environment. We provide a description of the basic 
theory, of the development phases and of the operation of 
the complete system. We also present preliminary results 
illustrating the processing of the data and supporting of the 
feasibility of this approach. 

Index Terms — Motor learning. Space remapping, wearable 
sensors, assistive technology, virtual reality 



I. Introduction 

ROBOTICS may be exploited to assist people in a 
great variety of activities [l]-[4]. Elderly and dis- 
abled people, in particular, are likely to benefit from 
these new technologies [3], [5], [6]. As they become 
limited in their mobility, they gain a greater degree 
of independence through the use of assistive devices 
such as powered wheelchairs(Fig. |l|. However, loss of 
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coordination and cognitive impairments can render dif- 
ficult or impossible to execute steering maneuvers, with 
consequent fatigue, frustration, reduced social life and 
risks of dangerous accidents. One way to overcome 
these difficulties is to equip the chair with an intelligent 
controller, sharing planning and execution of actions 
with the user. This cooperation between human and 
machine can be compared to the cooperation between 
a horse and its rider: the rider navigates (global plan- 
ning, ride control), while the horse avoids obstacles and 
makes path adjustments (fine motion control). A different 
approach - pursued here - is to allow the users to control 
the vehicle's motions at all levels. This second approach 
requires establishing a rapid communication between 
human and machine. However, one of the most chal- 
lenging tasks does not concern the technology of com- 
munications and control, but rather the reorganization 
of movements and the development of new cognitive 
maps of motor space. This is something most of us are 
familiar with, as we learn to drive a car. At first, the 
controls are foreign objects that require constant focus 
and attention. But, as we become expert drivers, the car 
becomes an extension of our bodies and the acts that 
we perform on the steering wheel and the pedals are 
directly and seemlsessly mapped into into their spatial 
and temporal consequences. Here, we plan to achieve 
the same result in disabled populations through the 
interaction of human and machine learning. This paper 
desctibes the basic platform, which includes wearable 
sensing, interfacing and VR technologies. 

A typical powered wheelchair [7]-[12] is operated by 
two rear differential and two front castor wheels. Two 
high torque motors drive the rear wheels. Most powered 
wheelchairs come with a programmable joystick to drive 
and operate it. The joystick controller has four directional 
commands i.e. forward, backward, left and right and 
a zero position to halt the wheelchair operations. The 
velocity of the wheelchair incrementally increases up 
to a fixed limit by holding the joystick continuously 
in the desired direction. While the joystick is a simple 
control device, it still represents a fixed interface that 
the user must learn to operate by mapping joystick 
into wheelchair motions. Accidents are often caused by 
the insufficient training on the handling of the joystick 
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Fig. 1. System concept. The virtual environment provides a safe training platform, where the control parameters are set according to the motor 
skills of the users. Once a satifactory behavior is reached, the control parameters will be applied to an actual powered wheelchair. 



and on the proper control procedures. Moreover, the 
wheelchair's training itself is a dangerous process, es- 
pecially for spinal cord injured users. 

The need to apply learning technologies to control 
assistive devices is highlighted by a recent survey on 
the use of powered wheelchairs [13]. The authors inter- 
viewed 200 clinicians in spinal cord injury facilities, reha- 
bilitation centers and geriatric care facilities. They asked 
about wheelchair user's feedback on the performance 
of different control interfaces, such as the joystick, sip- 
and-puff systems and head-an-chin devices. The study 
showed that about 10 percent of the disabled users "find 
it extremely difficult or impossible'' to use the wheelchair 
while 40 percent of the users report difficulties in steer- 
ing and maneuvering tasks. It is noteworthy that these 
figures refer to users that received specific (although 
conventional) training for controlling the wheelchair. In 
light of these difficulties, our approach is based on two 
characteristic features of the sensorimotor system: 

• Its ability to adapt to changes in the environment 
and 

• Its ability to exploit a large number of degrees of 
freedom for carrying out a variety of tasks. 

We exploit these features for designing a body-machine 
interface that will allow disabled users to operate a 
variety of devices [4], [14]-[18]. In particular, we will aim 
at creating a learning and design framework for spinal 
cord injured people with complete injuries at the C5-6 
cervical level, or incomplete injuries in the cervical cord. 
These injuries result in tetraplegia with limited residual 
body motions. 



II. Overall Methodology and System 
Description 

This article describes a novel method for controlling 
a powered wheelchair by spinal cord injured people. 
A wearable sensor shirt which is adequate to detect 
upper body (wrist, elbow and shoulder) movements, is 
custom built to extract some residual body movements 
of the users. A combination of virtual reality and signal 
processing methods is used for developing an effective 
body/ device interface and for carrying out training pro- 
cedures. The proposed system architecture is sketched in 
Fig. 1 and is based on four modules: 

proposed system architecture is sketched in Figll] and 
is based on three modules: 

• Sensor Shirt: The sensor shirt is composed of 52 
piezoresistive sensors that detect local fabric defor- 
mation caused by the movement of the user's upper 
body (i.e. wrist, elbow and shoulder; see section [lll|. 

• Data Acquisition and Signal Processing: Signals 
acquired from the sensors are processed and the 
control parameters for the wheelchair control are 
determined (Section [ill- A| ). 

• Virtual Reality: Preliminary system tests are per- 
formed on a virtual reality simulator of a powered 
wheelchair (section [IV]). The patient is trained to 
execute maneuvers of variable complexity, such as 
navigating a desert scene, moving among obstacles 
and following other moving objects. 

• Human ControlU sers are immersed in the PARIS 
system. After an initial calibration (Section V) 
they begin practising the control of the simulated 
wheelchair. The signals generated by the shirt are 
transformed in command variables, which are in- 
tegrated and combined with a head tracker (Flock 




Fig. 2. Sensor Shirt front view 




Fig. 3. Sensor Shirt back view 



of Birds, Ascension Technology [19]-[21]) to gen- 
erate the current viewpoint from the simulated 
wheelchair. 

III. Sensor Shirt 

The sensors of the shirt (Figures |2J [3] and [4l are made 
of a conductive elastomer (CE) material (commercial 
product provided by Wacker LTD [22]) printed on a 
Lycra /cotton fabric previously covered by an adhesive 
mask. 

CE composites show piezoresistive properties when a 
deformation is applied [23]. CE materials can be applied 
to fabric or to other flexible substrate, they can be em- 
ployed as strain sensors [24], [25] and they represent an 
excellent trade-off between transduction properties and 
possibility of integration in textiles. Quasi-statical and 
dynamical sensor characterization has been done in [24]. 
CE sensors exhibit some non-linear dynamical properties 
and relatively long relaxation times [26], [27] which 
should be taken into account in the control formulation. 




Fig. 4. a) Sensors on the back portion of the shoulder, b) Sensors on 
the muscle arm joint, elbow and wrist. c)Front view of the sesnors 
covering front shoulders and limb area. 
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(b) 



(c) 



Fig. 5. a) Virtual reality scene of our floor plan including corridors 
and small rooms which is projected in the PARIS. b)Robotics Virtual 
wheelchair is navigating through a door way. c)Patient is driving the 
Robotics wheelchair on the marked path inside the virtual reality. 



A. Signal Acquisition 

The analog signals acquired from the sensors are 
amplified and then digitized using a general purpose 64 
channels acquisition card and real-time processed using 
a personal computer. Real-time signal processing has 
been performed by using the xPC-Target(R)toolbox of 
Matlab®. The output of the signal processing stage, i.e 
the wheelchair controls, are sent to the virtual wheelchair 
described in the section below by using UDP connection. 

IV. Virtual Wheelchair and Personal 
Augmented Reality Immersive System (PARIS) 

A. Software 

The virtual wheelchair and its surrounding environ- 
ment are designed using VRCO's CAVELib ^^3D Graph- 
ics [28], CoinSD graphics libraries and VRML models 
[29]. The whole program was simulated on a Personal 
Augmented Reality Immersive System (PARIS) as de- 
scribed in [30] and [31]. PARIS provides the user with a 
perspective view of the scene. By wearing the specially 
designed goggles and head-tracker, users observe the 
scene from the viewpoint of the moving wheelchair. The 
goggles are actively switched and synchronized with 
the projection system to provide 3D stereo vision of 
the artificially generated images. The scene is updated 
asynchronously, based on an external input from the 
sensor shirt and from a head mounted 3D tracker. Fig. [5] 
shows a layout for the virtual environment, composed 
of several corridors, walls, obstacles and doorways. A 
path (represented by the white track of Fig|5| is drawn 
on the floor as guide for the subject to track during the 
learning phase (Section |Vl|. 



B. Virtual Wheelchair Kinematics Model 

The wheelchair is modeled as a simple two-wheel 
vehicle [32], [33], as shown in Fig. [6] The non-holonomic 




Fig. 6. a) Virtual wheelchair kinematics model based upon unicycle 
robot, b) Virtual wheelchair's 3D model created using Coin3d [29] 
libraries. 




Fig. 7. Virtual wheelchair's position update. 

kinematic equations of the wheelchair are: 

x{t) = v(t)cos{e{t)) 
y{t) = v(t)sin{0{t)) 
0{t) = uj(t) 



(1) 



The kinematic model of the wheelchair has two inputs, 
the translational velocity, v and the rotational velocity 
{(jj). In discrete time, the wheelchair's laws of motion 
are: 



Xk+i = ^/c + VkCos{Ok)At 
Vk+i =yk^ Vksin{ek)At 



(2) 



The two control inputs, ui and U2r are generated by pro- 
cessing algorithms applied to the shirt signals (Section 
V-A[ ). The virtual wheelchair position update from point 
{xk.Vk) to point (xfe+i,7/fe+i) (Fig.[7]) is given by: 



AS = VkAt = uiVfAt 

AO = UJkAt = U2VrAt 



(3) 
(4) 



where Vr and Vf are the maximal rotational and for- 
ward velocities, respectively, and At is the time interval 
between the two consecutive frames of the PARIS. 

V. Mapping body movements into Virtual 
Wheelchair (PARIS based) controls 

As a first stage in forming a map from body move- 
ments to wheelchair control, the nature of the control 




(a) 




(b) 

Fig. 8. (a) Right elbow signals extracted during user elbow flexion, 
(b) First principal component extracted from the raw signals retaining 
the 80% of the variance. 



needs to be determined. The controls ui and U2, for 
example, may specify the translational velocity v and 
rotational velocity (cj). Alternatively, the controls may 
specify accelerations v and Co instead of velocities. Given 
that we would like to allow the patient to remain in a 
fixed and comfortable position as much as possible we 
suggest to map one of the controls to the linear acceler- 
ation of the wheelchair. This allows the patient to cruise 
at a fixed velocity while maintaining the resting posture. 
Unlike the transitional velocity, the rotational velocity 
would typically be maintained at zero and would set to 
nonzero values for only short periods of time. For that 
reason we decided to map the second control signal to 
the rotational velocity. In order to map the shirt signals to 
the controls one needs to assign certain body movements 
to each control. This allows for great flexibility, as the 
vocabulary is determined by the users, based on their 
specific movement ability and personal preferences. In 
our preliminary experiments, our subject decided to use 
the following body movements: 

• Right elbow flexion was used to increase the value 
of Ui. 

• Left elbow flexion was used to decrease the value 
of Ui. 

• Right shoulder movement forward (scapular pro- 
traction) was used to increase the value of U2. 

• Left shoulder movement forward (scapular protrac- 
tion) was used to decrease the value of U2. (see Fig. 

Since the shirt contains several sensors at each joint, we 
examined the possibility of reducing the dimensionality 
of the shirt signals by applying Principal Component 
Analysis (PCA) [34]-[37] to the signals originating from 
the same joint. PCA was performed on data that were 
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Fig. 10. (a)Rectified Derivative Algoritlim. (b)Control scheme block 
diagram. 



Fig. 9. Virtual Robotics Wheelchair's position update w.r.t. the body 
movements. . . 

collected while the subject was moving his arms and 
shoulders in an uninstructed manner for a period of 10 
seconds. We found that the first principal component 
(PC) of each joint captures 80% — 90% of the same-joint- 
sensors variance (see Fig.|8]). Thus, for the above control 
scheme, which was chosen by the subject, we use four 
signal combinations, the first PC of the right shoulder 
(hrs), the first PC of the left shoulder (his), the first PC 
of the right elbow (hre), and the first PC of the left elbow 

(h,e). 




Fig. 11. Rectified derivative algorithm example. 



A. Description of Algorithms 

For removing possible drift and noise artifacts from 
the shirt signals, we used the following algorithm. The 
time derivative of each of the four PC's was calculated 
and a dead-zone was applied to each of them. The 
signals were then positive-rectified, as we are only in- 
terested in the rising part of each PC ((see Fig. p^a)). 
An example for the operation of the algorithm is shown 
in Fig. 11 The processed signals from the two elbows are 
then subtracted from each other to generate the transi- 
tional acceleration, while the processed signals from the 
two shoulders are subtracted from each other to generate 
the rotational velocity (see Fig. p^b)). 



VL Experimental Results 

A. Experimental Setup 

We present the results of a preliminary study con- 
ducted on a consenting adult participant approved by 
Northwestern University's Institutional Review Board 
(IRB). The participant wore a shirt, embedded with 52 
piezo-resistive sensors, capable of detecting the wearer's 
residual mobility. With the shirt on, the subject was 
seated in front of a virtual reality system (discussed in 
section [lll| & [Tv|. The virtual scene depicted in Figp^c) 
was modelled on a generic building floorplan, with mul- 
tiple rooms, doors and corridors. A thick white line was 
marked on the floor and the subject was asked to navi- 
gate through the corridors and doorways following the 
white track Fig. 13 ^d). The subject was able to navigate 



in the environment with little practice using arm and 
shoulder movements. Fig. 13 a) shows the trajectory of 
the virtual wheelchair (red line) as the subject attempted 
to track the pathway (blue line). The raw shirt signals, 
extracted principal components and relative controls for 
the trajectory experiment are shown in Fig. p^c). 



Trajectory 



B. Trajectory Analysis 

After each trial, the trajectories obtained from the 
participant's body movements were plotted against the 
prescribed path. It is important to note here that all of the 
trajectories were obtained using a uniform sensor shirt 
control scheme. We analyzed the trajectories using the 
following measures: 

1) The distance travelled by the subject from the start 
point to the end point of the prescribed path called 
(Dist) (shown in figl2 a,b)). 

2) The error between the prescribed trajectory and the 
subject's actual trajectory obtained by calculating 
the segmented area between both trajectories, from 
the start point to the end point of the prescribed 
path called {Ediff). 



In the first trial result shown in fig 13 a) the subject 
began by familiarizing himself with the control strategy 
(through arm and shoulder movements) without follow- 
ing the prescribed trajectory. When the subject completed 
this initial step he started following the prescribed path 
(also shown in [l3ta)). The participant moved in differ- 
ent directions in the virtual environment as shown in 



fig 13 a), to learn the control criteria. As the subject spent 
more time moving in the virtual scene the understanding 
of the control map improved and the subject was able 
to navigate the scene with greater accuracy. 

The data in fig[T4|c,d) shows a monotonic reduction 
in the subject's trajectory error from trial to trial. This 
is consistent with the hypothesis that, through practice, 
a subject is able to adapt to their environment using 
the novel control strategy of moving a wheelchair with 
shoulder and arm movements. The decreasing error 
trend is evident for both Ediff and dist over all of the 
trials. 

The results in fig[l4];a,b,c&d) plot the total distance 
traveled by the subject for each trial to reach the pre- 
scribed endpoint from the starting point. The drastic 
reduction in area and distance error between the first, 
second and third trials, shows that the subject's initial 
mobility adjustments are significant. In subsequent trials, 
the subject's movement adjustments are more finely 
tuned as the subject's familiarity of the sensor shirt- 
wheelchair control plan improves resulting in smaller 
distance errors. 

VII. CONCLUSION 

The combination of robotics technology, intelligent 
interfaces and virtual reality allow us to develop new ap- 
proaches to the design of assistive devices. Our approach 
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Fig. 12. (a) Subject trajectory obtained from one of the experiments. 
The black circles show the intersection points of both trajectories (b) 
The subject trajectory is segmented area wise (i.e. in portions, after 
considering the points of intersections), inorder to calculate the error 
area, (c) Principal components of shirt's raw signals responsible to 
produce control signals neccessary for navigating the wheelchair in 
the virtual reality environment. 
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Fig. 13. In (a) red trajectory is obtained on the first day of experiment wlien subject used the sensor shirt and was asked to move on the 
marked line (in blue), (b) Shows all trajectories (in red) obtained after 22 preliminary experiment obtained from the subject's travel on the 
marked line in the immersive virtual environment. In (c) subject sitting infront of Virtual Reality is immersed in the 3D scene and navigating 
the wheelchair by the residual mobility captured by sensors, along the line marked on the floor. In (d) the top view of the virtual environment 
is shown with the line (blue coloured) marked on the floor. 
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Fig. 14. (a) Distance reduction in the trajectories after every trial, (b) Area error measure {Ediff) between the prescribed and subject's trajectory 
at each time per day trial. 



is based on the key concept that the burden of learning 
should not fall entirely on the human operator. The field 
of machine learning has been rapidly developing in the 
recent decade and is now sufficiently mature to design 
interfaces that are capable of learning the user as the user 
is learning to operate the device. In this case, "learning 
the user'' means learning the degrees of freedom that the 
user is capable to move most efficiently and mapping 
these degrees of freedom onto wheelchair controls. We 
should stress that such mapping cannot be static, as 
in some cases the users will eventually improve with 
practice. In other, more unfortunate cases, a disability 
may be progressive and the mobility of the disabled 
user will gradually deteriorate. In both situations the 
bodymachine interface must be able to adapt and to 
update the transformation from body-generated signals 
to efficient patterns of control.The final aim is to facilitate 
the formation of new and efficient maps from body 
motions to operational space. 
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